Robust Video Coding. Heechan Park. Signal and Image Processing Group Computer Science Department University of Warwick. for CS403

Robust Video Coding for CS403 Heechan Park Signal and Image Processing Group Computer Science Department University of Warwick

Standard Video Coding Scalable Video Coding Distributed Video Coding

Video Coding Compression! Video recording with 1M pixel cam [ 1 Mega pixel x 1Byte x 30frame ] / sec

frame 1 frame 2 frame 7 JPEG coding for each frame? Not efficient way, only slight change frame to frame

How to compress video Motion estimation / compensation Residual Coding

Sender ----> Receiver Encoding Channel Decoding

Sender ----> Receiver Encoding Channel Decoding current frame

Sender ----> Receiver Encoding Previous frame Channel Decoding current frame

Sender ----> Receiver Encoding Previous frame Channel Decoding Motion Estimation / Compensation current frame

Sender ----> Receiver Encoding Previous frame Channel Decoding Motion Estimation / Compensation Prediction current frame

Sender ----> Receiver Encoding Previous frame Channel Decoding Motion Estimation / Compensation Prediction current frame Closed loop system

Motion Estimation Previous Frame sender/receiver Current Frame sender

Motion Estimation Mean Absolute Difference Previous Frame sender/receiver Current Frame sender

Motion Estimation Mean Absolute Difference copy & paste Previous Frame sender/receiver Current Frame sender

Mesh-Based ME

Mesh-Based ME Triangle deformation is expressed by Affine Transform

Residual Coding

Residual Coding w/o ME with ME

Residual Coding w/o ME with ME (DCT Wavelet) > Quantisation > Entropy

H.264 / MPEG-4 AVC Latest Standard, Finely tuned over decades succeeding H263 and MPEG4 Offers 50:1 compression ratio (J2K 40:1) Bluray / HD-DVD H264 = great so what is next now?

Compactness

Robustness Compactness

Compactness Robustness Scalability

What is problem in this system? Encoding Decoding

What is problem in this system? Encoding Decoding Wireless Noisy Channel

What is problem in this system? Encoding Decoding Wireless Noisy Channel Drifting Error Accumulates

What is problem in this system? Encoding Decoding Wireless Noisy Channel Drifting Error Accumulates Target bitrate is fixed; not flexible to cope with variable connection

Scalable Video Coding

Wavelet Temporal Filtering

Wavelet Temporal Filtering End-user can choose right resolution for his connection

Wavelet Temporal Filtering End-user can choose right resolution for his connection alleviates drifting errors

Wavelet Temporal Filtering End-user can choose right resolution for his connection alleviates drifting errors but not a perfect solution for channel noise

Distributed Video Coding

1970 Slepian and Wolf Independent encoding Joint decoding Girod IEEE Video Coding and Delivery 2005

1970 Slepian and Wolf Independent encoding Joint decoding X and Y are correlated Girod IEEE Video Coding and Delivery 2005

1970 Slepian-Wolf Theorem RX H(X Y) RY H(Y X) (RX + RY) H(X, Y) Girod IEEE Video Coding and Delivery 2005

Later, Wyner and Ziv established the min rate to encode X guaranteeing its recon. when Y is already known to decoder but not to encoder decoder aware of the correlation between X and Y No rate increase when X and Y are jointly Gaussian Girod IEEE Video Coding and Delivery 2005

Later, Wyner and Ziv X, Y correlated established the min rate to encode X guaranteeing its recon. when Y is already known to decoder but not to encoder decoder aware of the correlation between X and Y No rate increase when X and Y are jointly Gaussian Girod IEEE Video Coding and Delivery 2005

Later, Wyner and Ziv X, Y correlated X=Y established the min rate to encode X guaranteeing its recon. when Y is already known to decoder but not to encoder decoder aware of the correlation between X and Y No rate increase when X and Y are jointly Gaussian Girod IEEE Video Coding and Delivery 2005

Later, Wyner and Ziv X, Y correlated Channel coding X=Y established the min rate to encode X guaranteeing its recon. when Y is already known to decoder but not to encoder decoder aware of the correlation between X and Y No rate increase when X and Y are jointly Gaussian Girod IEEE Video Coding and Delivery 2005

Later, Wyner and Ziv X, Y correlated X=Y Error Correction Channel coding established the min rate to encode X guaranteeing its recon. when Y is already known to decoder but not to encoder decoder aware of the correlation between X and Y No rate increase when X and Y are jointly Gaussian Girod IEEE Video Coding and Delivery 2005

Example standard video coding Correlation = hamming distance is at most 1 Given Y = [ 0 1 0 ] Encode X = [ 0 1 1 ]

Example standard video coding Correlation = hamming distance is at most 1 Given Y = [ 0 1 0 ] Encode X = [ 0 1 1 ] X = [ 0 1 0 ] or [ 1 1 0 ] or [ 0 0 0 ] or [ 0 1 1 ]

Example standard video coding Correlation = hamming distance is at most 1 Given Y = [ 0 1 0 ] Encode X = [ 0 1 1 ] X = [ 0 1 0 ] or [ 1 1 0 ] or [ 0 0 0 ] or [ 0 1 1 ] 4 possibilities; Need 2 bits to encode

Example distributed video coding Correlation = hamming distance is at most 1 Given Y = [ 0 1 0 ] Encode X = [ 0 1 1 ]

Example distributed video coding Correlation = hamming distance is at most 1 Given Y = [ 0 1 0 ] Encode X = [ 0 1 1 ] Coset 1 Coset 2 Coset 3 Coset 4 [ 0 0 0 ] [ 1 1 1 ] [ 0 0 1 ] [ 1 1 0 ] [ 0 1 0 ] [ 1 0 1 ] [ 1 0 0 ] [ 0 1 1 ]

Distributed Video Coding Encoding Channel Decoding

Distributed Video Coding Encoding Channel Decoding partial info for each block

Distributed Video Coding Encoding Channel Decoding partial info for each block refresh each block with partial info -> no drift error

Coset code and Side info(y) Source Side Info Ramchandran ICIP 2005

Quantised DCT coefficient Generating Cosets

Generating Cosets Quantised DCT coefficient x

Generating Cosets Quantised DCT coefficient x coset 1 coset 2

Generating Cosets Quantised DCT coefficient x coset 1 coset 2 y

Motion Estimation Previous Frame Current Frame

Motion Estimation Too many side info(y) Previous Frame Current Frame

Motion Estimation Too many side info(y) CRC or Hash Previous Frame Current Frame

Motion Estimation Too many side info(y) copy & paste CRC or Hash Previous Frame Current Frame

Packet Loss Experiment H.264 + FEC SVC + DVC Ramchandran PCS 2004

Packet Loss Experiment H.264 + FEC SVC + DVC Separate source coding and channel coding Ramchandran PCS 2004

Packet Loss Experiment H.264 + FEC SVC + DVC Separate source coding and channel coding Joint source / channel coding Ramchandran PCS 2004

Pereira, MPEG Future Video Coding Workshop, Nice, France 2005

Standard Video Coding Distributed Video Coding Encoding Slow Very Fast Decoding Fast Slow Power Consumption High in encoder / Low in decoder Low in encoder / High in encoder Motion Estimation in encoder in decoder Compression ratio 50:1 (60:1) ~30:1 Noise resilient No Yes Applications DVD, Streaming from server,... Streaming from mobile, Sensor Network,...

References Girod, A. Aaron, S. Rane and D. Rebollo-Monedero, "Distributed video coding," Proceedings of the IEEE, Special Issue on Video Coding and Delivery, vol. 93, no. 1, pp. 71-83, January 2005 R. Puri and K. Ramchandran, PRISM: A new robust video coding architecture based on distriubted compression principles, in Proc. Allerton Conference on Communication, Contro, and Computing, Allerton, IL, Oct. 2002 S. S. Pradhan and K. Ramchandran, Generalized Coset Codes for Distributed Binning, IEEE Trans. Information Theory, Vol. 51, No. 10, Oct. 2005 M. Tagliasacchi, A. Majumdar and K. Ramchandran, A Distributed- Source-Coding Based Robust Spatio-Temporal Scalable Video Codec, Picture Coding Symposium 2004

Conclusion Conventional video coding structure and its problem Scalable Video Coding Distributed Video Coding Joint source-channel coding : Wyner - Ziv coding Still very early stage and more works to be done for better coding efficiency

Questions?