By :- Ramolia Pragnesh R. Guided by :- Dr. K.R.Rao Dr. Dongil Han Term :- Fall-2009 1
Introduction to AVS-M Overview of AVS-M Complexity present in AVS-M encoder Various approaches to reduce complexity Introduction to data mining algorithm: C4.5 Project implementation steps AVS-M execution, and mode and attribute extraction. Future work. 2
Introduction to AVS-M AVS-M is the seventh part of video coding standard developed by AVS workgroup of China which aims at mobile applications. It has 9 different levels for different formats. It supports only progressive video coding hence codes frames only. It uses only 4:2:0 chroma sub-sampling format. It uses only I and P frames. 3
Different parts of AVS [10] Part Name 1 System 2 Video 3 Audio 4 Conformance test 5 Reference software 6 Digital media rights management 7 Mobile video 8 Transmit AVS via IP network 9 AVS file format 10 Mobile speech and audio coding Table 1: Different parts of AVS 4
Layered Data Structure Sequence Picture Slice Macro Block Block G.O.P. Sequence Picture Slice Block Macro block 5
AVS-M Codec Each MB needs to be intra or inter predicted. Switch S0(Fig. 1 ) is used to decide between inter and intra based type of MB. Unit size for intra prediction is block size of 4x4, and predictions are derived based on left and upper blocks. Inter predictions are derived on blocks of varying sizes: 16x16, 16x8, 8x16, 8x8, 8x4, 4x8, and 4x4 from locally reconstructed frames. Transform coefficients are coded by VLC. Deblocking filter is applied on reconstructed image. 6
Encoder Figure 1: Encoder of AVS-M [10] 7
Decoder Figure 2: Decoder of AVS-M [10] 8
Major and Minor tools of AVS-M Network abstraction layer (NAL). Supplemental enhancement information (SEI). Transform 4x4 integer transform. Adaptive quantization of step size varying from 0-63. Intra prediction 9 modes (Fig. 3), simple 4x4 intra prediction and direct intra prediction. Motion compensation 16x16, 16x8, 8x16, 8x8, 8x4, 4x8, and 4x4 block sizes. Quarter-pixel interpolation 8-tap horizontal interpolation filter and 4-tap vertical interpolation filter. Simplified in-loop de-blocking filter. Entropy coding. Error resilience. 9
Intra adaptive directional prediction [25] Figure 3: Intra adaptive directional prediction 10
Intra prediction Intra prediction scheme in AVS-M brings much simplicity as compared to H.264 baseline profile. It uses 4x4 block as the unit for intra-prediction. It uses 2 modes of prediction in intra prediction intra_4x4 and direct intra prediction. Intra_4x4 uses content based most probable intra mode decision as shown in Table 2 to save bits, where U and L represents the upper ad left blocks as shown in Fig. 4. Upper block[u] Left block[l] Current block Fig. 4 : Current block and neighboring block representation Direct intra prediction brings much of the compression based on trade-off decision. 11
Intra prediction U L -1 0 1 2 3 4 5 6 7 8-1 8 8 8 8 8 8 8 8 8 8 0 8 0 0 2 0 0 0 2 0 2 1 8 2 1 2 2 2 2 2 2 2 2 8 2 2 2 2 2 2 2 2 2 3 8 2 1 2 3 4 5 2 7 2 4 8 4 4 2 4 4 4 6 4 4 5 8 5 5 2 5 5 5 6 5 5 6 8 6 6 6 6 6 6 6 6 6 7 8 7 7 2 7 7 7 6 7 7 8 8 0 1 2 3 4 5 6 7 8 Table 2: Content based most probable mode decision table [25] Mode -1 is assigned to L or U when the current block does not have Left or Upper block respectively. 12
Inter-frame prediction Size of the blocks in inter-frame prediction can be 16x16, 16x8, 8x16, 8x8, 8x4, 4x8, and 4x4 depending on the amount of information present within the macro-block. Motion is predicted up to ¼ pixel accuracy. If the half_pixel_mv_flag is 1 then it is up to ½ pixel accuracy. 8-tap filter F1 = ( 1,4, 12,41,41, 12,4, 1) and 4-tap filter F2 = ( 1,5,5, 1) are used for horizontal and vertical interpolations respectively for ½ pixel MV search and averaging (liner interpolation) is used for ¼ pixel accuracy as shown in Fig. 6. 13
Inter frame block sizes: 7 block sizes are present in AVS-M for inter frame prediction [9]. 4 4 4 4 0 0 1 4 4 0 1 0 1 2 3 4 4 8 8 8 8 8 8 8 8 0 16 0 1 8 8 8 8 0 1 16 8 8 0 1 2 3 8 8 16 16 Figure 5: Inter frame prediction block sizes 14
sub-pixel motion estimation by interpolation[16]: Figure 6: interpolation of sub-pixels (hatched lines show half-pixels, empty circles are quarter-pixels, and capital letters represent full-pixels.) 15
Error concealment and resilience 3- techniques are used for error concealment forward, backward and interactive error concealment. For error resilience supplemental enhancement information (SEI) is sent along with the bit-stream which has details of 1) The frame number from which particular block motion starts and 2) Type of motion (zooming out/in, transversal motion in plane etc.). SEI helps to recover any information lost due to transmission error. 16
AVS-M encoder complexity variable block sizes in Inter Mode. It supports 9 intra_4*4 mode and 1 Direct_intra prediction mode. Full search for motion estimation gives the optimum result, but that comes along with implementation complexity. For example, assuming FS(full search) and M block types, N reference frames and a search range for each reference frame and block type equal to +/- W, we need to examine N x M x (2W + 1)^2 positions compared to only (2W + 1)^2 positions for a single reference/block type. 17
Continued 7 inter prediction modes because of 7 different block sizes, 9 intra_4*4 modes and 1 direct intra prediction mode. ¼ and ½ pixel accuracy in motion vector estimation. 18
Various techniques to reduce complexity Intra mode selection algorithm[26]. Only intra spatial-prediction scheme[27]. Fast mode decision algorithm for intra prediction for H.264/AVC [28]. Dynamic control of motion estimation search parameters for low complexity H.264[29]. Adaptive algorithm for fast motion estimation [30]. Data mining algorithm for fast motion estimation [2]. 19
Data mining algorithm C4.5 Extracts information from data automatically, by computational and statistical methods. Based on the information extracted, develops trees. These trees give the decision statement for mode decision Takes decision based on metrics such as MB mean, MB variance, amplitude of edge detection, residual variance etc. 20
Goal of this project:- Implement data mining algorithm c4.5 to decide the inter prediction block mode. 21
Implementation steps:- Select number of frames of a video sequence in QCIF as training sequences. Obtain the required attributes off line Encode the training sequence using full complexity AVS-M encoder Store the attributes calculated off line and mode decision taken by encoder in ARFF file Feed this ARFF file to weka tool, which will give decision tree similar to that of figure 10. 22
Continued Mask the motion estimation part in the actual AVS-M encoder Overwrite that with if-else statements based on the decision tree Compare the performance of the simple codec with actual AVS-M 23
AVS-M execution and mode &attribute extraction: 24
Input parameters defined parameter Akiyo_qcif Akiyo_qcif Foreman_qcif Foreman_cif Frame size 176*144 176*144 176*144 352*288 No. of frames coded 60 60 60 60 Intra period 20 0 0 0 QP_first frame 28 28 28 28 QP_rest frames 40 40 40 40 Frames/sec 30 30 30 30 Output file name test.avs test.avs test.avs test.avs Recon file name Test_rec_.yuv Test_rec_.yuv Test_rec_.yuv Test_rec_.yuv Table 3: Parameters defined in encoder.cfg 25
Encoder performance: Parameter Akiyo_qcif Akiyo_qcif Foreman_qcif Foreman_cif Original file size 2227.5 Kbytes 2227.5 Kbytes 2227.5 Kbytes 8910 Kbytes Encoded file size 20833 bytes 15376 bytes 54069 bytes 172468 bytes Reconstructed file 2227.5 Kbytes 2227.5 Kbytes 2227.5 Kbytes 8910 Kbytes size Decoded file size 2227.5 Kbytes 2227.5 Kbytes 2227.5 Kbytes 8910 Kbytes Compression Ratio 109.487 : 1 148.34 : 1 42.168 : 1 52.90 : 1 SNR(Y) 38.43 db 38.74 db 37.31 db 37.33 db SNR(U) 40.20 db 40.33 db 41.05 db 40.99 db SNR(V) 40.98 db 41.21 db 42.15 db 43.31 db SNR (YUV) 39.039 db 39.31 db 38.325 db 38.74 db Encoding time 142.08sec. 145.127 sec 211.07sec 903.202 sec Decoding time 12.391 sec 20.54 sec 22.94 sec 171.87 sec Bit rate 83.9295 Kbps 61.105Kbps 215.85Kbps 689.87Kbps Table 4: AVS-M performance 26
Encoder output for foreman_qcif sequence Figure 7: AVS-M output sceenshot 27
AVS-M decoder output: Figure 8: AVS-M decoder screenshot 28
AVS-M mode decisions: Figure 9: AVS-M mode decisions extracted 29
Encoded and decoded frame: a. Original Akiyo sequence b. Reconstructed Akiyo sequence c. Decoded Akiyo sequence Fig. 10: 45 th frame: a. original frame b. reconstructed frame on the encoder side c. Decoded frame on the decoder side 30
Original, reconstructed and decoded foreman_cif frame: Figure 11:50 th frame: a. original frame b. reconstructed frame on the encoder side c. Decoded frame on the decoder side 31
Original, reconstructed and decoded foreman_qcif frame: Figure 12:50 th frame: a. original frame b. reconstructed frame on the encoder side c. Decoded frame on the decoder side 32
.arff file: Figure 13:.arff file look-how 33
Further plan: 1) Get a decision tree from weka tool for attributes: mean, variance, and edge vector with mode decision as class. 2) Embed the c++ code to extract attributes into AVS-M to extract attributes on line for all the test sequences. 3) Mask the motion estimation part in AVS-M and implement the decision tree obtained in step-1. 34
Example of the decision tree generated by C4.5 Figure 13: Decision tree generated by weka tool 35
References: [1]http://ee.uta.edu/Dip/Courses/EE5359/Multimedia%20Processing%20 project%20report%20final.pdf ; course website UTA [2]P. Carrillo, H.Kalva and T.Pin Low complexity H.264 video encoding", SPIE. vol.7443, Paper # 74430A, Aug. 2009 [3]Kusrini1, Sri Hartati2 Implementation of C4.5 algorithm to evaluate the cancellation possibility of new student applicants at STMIK AMIKOM YOGYKARTA, Proceedings of the International Conference on Electrical Engineering and Informatics Institute Teknologi Bandung, Indonesia June 17-19, 2007 [4]S. Saponara, et al Adaptive algorithm for fast motion estimation in H.264/MPEG-4 AVC, Proc. Eusipco2004, pp. 569 572, Wien, Sept. 2004 [5]Décisions tree basics : http://dms.irb.hr/tutorial/tut_dtrees.php [6]Weka tool software :http://www.cs.waikato.ac.nz/ml/weka/ 36
Continued [7]X. Jing and L. P. Chua, An efficient inter mode decision approach for H.264 video coding International Conference on Multimedia and Expo (ICME), pp. 1111-1114, July 2004. [8]Software download: ftp://159.226.42.57/public/avs_doc/avs_software [9]Power point slides by L.Yu, chair of AVS video : http://wwwee.uta.edu/dip/courses/ee5351/ispacsavs.pdf [10]L.Fan, Mobile multimedia broadcasting standards, ISBN: 978-0-387-78263-8, Springer US, 2009 [11]AVS working group official website, http://www.avs.org.cn [12]Test sequences can be downloaded from the site http://trace.eas.asu.edu/yuv/index.html [13]Y.Xiang et al., Perceptual evaluation of AVS-M based on mobile platform, Congress on Image and Signal Processing, 2008, vol. 2, Issue, pp76 79, 27-30 May 2008. 37
Continued [14]M.Liu and Z.Wei. A fast mode decision algorithm for intra prediction in AVS-M video coding, vol.1, ICWAPR apos; 07, Issue, 2-4, pp.326 331, Nov. 2007. [15]L.Yu et al., Overview of AVS-Video: Tools, performance and complexity, SPIE VCIP, vol. 5960, pp. 596021-1~ 596021-12, Beijing, China, July 2005. [16]L.Yu, S.Chen, J.Wang, Overview of AVS-video coding standards special issue on AVS, SP:IC, vol. 24, p. 247-262, April 2009. [17]Y.Shen, et. al., A simplified intra prediction method, AVS Doc. AVS- M 1419, 2004. [18]F.Yi, et al., An improvement of intra prediction mode coding, AVS Doc. AVS-M 1456, 2004. [19]L.Xiong, Improvement of chroma intra prediction, AVS Doc. AVS- M1379, 2004 38
Continued [20]X.Mao, et al., Adaptive block size coding for AVS-X profile. AVS Doc. AVS-M2372, 2008. [21]R.Wang, et al., Sub-pixel motion compensation interpolation filter in AVS, 2004 IEEE International Conference on Multimedia and Expo, 1:93-96, 2004. [22]F.Yi et al., Low-complexity tools in AVS Part 7, J. Computer Science Technology, vol.21, pp. 345-353, May. 2006 [23]W.Gao and T.Huang AVS Standard -Status and Future Plan, Workshop on Multimedia New Technologies and Application, Shenzhen, China, Oct. 2007. [24]W.Gao et al., AVS the Chinese next-generation video coding standard, National Association of Broadcasters, Las Vegas, 2004. [25] Z.Ma, et al., Intra coding of AVS Part 7 video coding standard, J. Computer Science Technology, vol.21, Feb.2006. 39
Continued [26] Jongho Kim et.al, H.264 Intra Mode Decision for Reducing Complexity Using Directional Masks and Neighboring Modes, PSIVT 2006, LNCS 4319, pp. 959 968, 2006. [27]Xin, Vetro, Fast Mode Decision for Intra-only H.264/AVC Coding, TR 2006-034 May 2006. [28]Pan et. al Fast Mode Decision Algorithm for Intraprediction in H.264/AVC Video Coding, IEEE Transactions On Circuits And Systems For Video Technology. Vol 15, No. 7, July 2005 [29]S. Saponara et. al Dynamic Control of Motion Estimation Search Parameters for Low Complex H.264 Video Coding, IEEE Transactions on Consumer Electronics, Vol. 52, No. 1, FEBRUARY 2006. [30]Cheng-Chang Lien, Chung-Ping Yu, A Fast Mode Decision Method for H.264/AVC Using the Spatial-Temporal Prediction Scheme, ICPR 2006 40