Summer Internship at STMicroelectronics, Greater Noida, India

Transcription

1 Summer Internship at STMicroelectronics, Greater Noida, India Thibault Lefeuvre, Ecole Polytechnique, France August 2015 Techniques of frame compression in ultra low power encoder Introduction At first, my work has consisted in reading articles and documents on H.264 and frame compression algorithms ([Ric10], [Zhe06], [Hoi06], [Qin10]) since my knowledge of the issue was very basic. I have then studied the different metrics currently used to compare and assess the similarities between colocated macroblocks ([Zho02] [Zho04], [Chu09]) which I thought would be at the heart of the project. After discussion, I have focused myself on the Intra-only profile along with PSkip mode decision and on early-skip prediction as [Ref] puts forth. The idea was to build a mathematical background to this solution which proved efficient but lacked a theoretical background. Following [Ref], I ve tried to design an algorithm assessing the similarities between co-located macroblocks without explicitly referring to the previous frame, thus saving important DDR costs. Algorithm The algorithm can be described as follows : Frame N is transformed, using an edge detection kernel, into a black and white image where white pixels are edges or corners and black pixels form the background. The matrix obtained is a sparse one. Macroblocks of frame N are stored as follow : if the current transformed MB contains white pixels (edges or corners), then it is stored using a sparse matrix compression algorithm ; if not, the average luma of the original MB is stored. The current MB is compared to the co-located MB of frame N 1 : if both contain white pixels, a SAD between the black and white macroblocks is computed and if it is below a parameter threshold1, the current MB is skipped, otherwise MB is encoded using Intra prediction ; if they are both identified as background (only black pixels), absolute difference between average luma is computed and if it is below a parameter threshold2, current MB is skipped, otherwise MB is encoded using Intra prediction ; eventually, if one MB contains white pixels but the co-located MB doesn t, the current MB is encoded using Intra prediction. Afterwards, the algorithm jumps to frame N + 1. As far as greyscale images are concerned, instead of storing N M pixel-macroblocks, i.e. N M bytes (assessing that one pixel is encoded on one byte), the algorithm will either store a real floating number or a sparse matrix which compresses on average between 90% to 95% the macroblock. Overall, the data required to asses the similarity between the macroblocks of two consecutive frames is much reduced. 1

2 Figure 1: Algorithm flowchart Results The algorithm was implemented in MATLAB. The intra-prediction mode was not taken into account : the sequences were analyzed and the macroblocks which could be skipped were detected. The output frames are therefore made up of original macroblocks (non-skipped macroblocks) and macroblocks from the previous frame (skipped macroblocks). To illustrate the algorithm, we show the process of reconstruction of two frames taken from two different sequences : the first sequence is a hand moving on a plan, and the second sequence is the face of a man moving in the foreground. The pictures at the top are the initial pictures of the sequence (two following frames). In the middle, the images have been transformed using a canny edge detector : only the macroblocks identified as background remain in grayscale, whereas the macroblocks with edges or corners are turned into black and white. Pictures in the bottom, show the results after computation. The parameters of the algorithm have a critical influence on the output, especially the thresholds used in the canny edge detector, and the thresholds used after computing SAD. A good choice of parameters allow both an efficient compression (i.e. a lot of macroblocks are skipped) and a robust visual quality (the distortion is minimum for the human vision system). On the other hand, a bad set of parameters can either lead to an effective compression along with a poor video quality or to a deficient compression along with an acceptable video quality. We then display a sequence of six frames showing two cars moving in the foreground. The original frames are (1)-(6) and algorithm has been processed on frames (a)-(f). The frames are made of = 1024 macroblocks (blocks of pixels where one pixel is coded on one byte). The array below shows the compression efficiency of the algorithm on the data required to assess similarities between co-located macroblocks. We have made a distinction between macroblocks compressed as backgrounds and macroblocks compressed as edges since the storage process is different. For a MB stored as background, 4 bytes are required instead of 256 (16 16) and for a MB stored as edges, the use of a sparse matrix compression algorithm can empirically save up to 95-99% of memory but we will consider 90% in order to give a lower bound estimation so that a MB compressed 2

3 in this way will be considered to be stored on 26 bytes. The percentage of compression is calculated as the total bytes required divided by the total bytes encoding the frame. Frame Macroblocks identified as edges Macroblocks identified as background Total bytes required Percentage of compression 96.93% 96.88% 96.69% 96.59% 96.59% 96.61% Artifacts and improvement Tests still need to be carried out with a larger database in order to assess the validity of the algorithm. Overall, simple shapes (faces, hands, cars,...) prove to be better processed by the algorithm than detailed areas (landscapes). Artifacts appear in regions where the edge detection algorithm fails to underline the structure of the image, either because it is too noisy, or because the parameters were not well chosen, or because of a lack of contrast. To solve this problem, the solution may be to identify another kernel which would be efficient in processing these specific areas. References [Chu09] Chun-Ling Yang, Rong-Kun Leung1, Lai-Man Po, Zhi-Yi Mai. An SSIM-Optimal H.264/AVC Inter Frame Encoder. IEEE, [Hoi06] Hoi-Ming Wong, Oscar C. Au, Andy Chang, Shu-Kei Yip, Chi-Wang Ho. Fast Mode Decision and Motion Estimation for H.264 (FMDME). IEEE, [Qin10] Qin Liu, Takeshi Ikenaga. Rate-Distortion Optimization Based Skip Mode Early Detection in H.264. International Symposium on Intelligent Signal Processing and Communication Systems, December [Ref] A Novel PSkip determination Algorithm for WiGig encoder. [Ric10] Iain E. Richardson. The H.264 Advanced Video Compression Standard, Second Edition. Wiley, [Zhe06] Zhenyu Wei, King Ngi Ngan. A Fast Macroblock Mode Decision Algorithm for H.264. IEEE, [Zho02] Zhou Wang, Alan C. Bovik. A Universal Image Quality Index. IEEE Signal Processign Letters, Vol. 9, No. 3, March [Zho04] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, Eero P. Simoncelli. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing, Vol. 13, No. 4, April

4 Figure 2: (a) First original frame, (b) Second original frame, (c) First frame processed, (d) Second frame processed, (e) Second frame where : black MBs are identified as background and skipped, white MBs are identified as edges or corners and skipped, grayscale MBs are not skipped (encoded with intra mode), (f) Second frame reconstructed : black and white MBs in (e) (skipped MBs) have been replaced with MBs from the first frame. The parameters used with the algorithm are : N = 15, σ = 1, threshold1 = 6, threshold2 = 6, threshbw = 7, threshcol = 10. 4

5 Figure 3: (a) First original frame, (b) Second original frame, (c) First frame processed, (d) Second frame processed, (e) Second frame where : black MBs are identified as background and skipped, white MBs are identified as edges or corners and skipped, grayscale MBs are not skipped (encoded with intra mode), (f) Second frame reconstructed : black and white MBs in (e) (skipped MBs) have been replaced with MBs from the first frame. The parameters used with the algorithm are : N = 15, σ = 1, threshold1 = 12, threshold2 = 8, threshbw = 50, threshcol = 10. 5

6 Figure 4: Frames (1)-(6) are original and algorithm was processed on frames (a)-(f). The parameters used with the algorithm are : N = 15, σ = 1, threshold1 = 15, threshold2 = 8, threshbw = 40, threshcol = 10. 6