Distributed Source Coding for Image and Video Applications. Acknowledgements

Transcription

1 Distributed Source oding for Image and Video Applications Ngai-Man (Man) HEUNG Signal and Image Processing Institute University of Southern alifornia 1 Acknowledgements ollaborators at US: Antonio Ortega Huisheng Wang aimu Tang Ivy Tseng Some materials taken from literatures: iong et al (Texas A&M) Girod et al (Stanford) Ramchandran et al (U Berkeley) Others 2 1

2 Outline Introduction Motivation Source coding with decoder side-information only Simple example to illustrate DS idea Application scenarios Basic information-theoretic results Slepian-Wolf Wyner-Ziv Practical encoding/decoding algorithm Role LDP based Applications Low-complexity video encoding Scalable video coding Flexible decoding, e.g., multiview video 3 Motivation 4 2

3 ompression with losed-loop Prediction (LP): E.g., MPEG, H.26 Z=- ^ Exactly same predictor ompute and send the difference/prediction residue (Z=-) to decoder Predictor : Motion-compensated predictor in neighboring frames o-located pixels in neighboring slices in volumetric image 5 Issues in losed-loop Prediction: High omplexity Encoding High complexity encoding Motion estimation time Some emerging applications require low complexity encoding, e.g., mobile video, video sensors Distributed source coding allows flexible distribution of complexity between encoder and decoder 6 3

4 Issues in losed-loop Prediction: Vulnerable to Transmission Error Error in predictor propagates to subsequent frames: drifting time error Distributed source coding allows exact reconstruction with transmission error 7 Issues in losed-loop Prediction: Lack of Decoding Flexibility Some emerging applications need to support multiple decoding paths, e.g., multiview video View View View Time Decoding path When users can choose among different decoding paths, it is not clear which previous reconstructed frame will be available to use in the decoding DS can support multiple decoding paths and address predictor uncertainty efficiently 8 4

5 Distributed Source oding 9 Distributed Source oding Lossless compression of random variable - Intra coding: R H() (i) LP (e.g. DPM): Z=- R H( ) (ii) DS: ~ P(,) Encoding does not require de-coupled from the encoded data ~ 10 5

6 Distributed Source oding Lossless compression of random variable - Intra coding: R H() (i) LP (e.g. DPM): Z=- R H( ) (ii) DS: ~ P(,) Encoding does not require de-coupled from the encoded data ~ 11 Distributed Source oding Lossless compression of random variable - Intra coding: R H() (i) LP (e.g. DPM): Z=- R H( ) (ii) DS: ~ P(,) Encoding does not require de-coupled from the encoded data ~ 12 6

7 (i) LP (e.g. DPM): Distributed Source oding Lossless compression of random - Access variable to exact value of - ompute the residue - Intra coding: R H() - Use entropy coding to achieve compression - =4 - Z=- =2 R H( ) -: Entropy oding 001 (ii) DS: -~ Do not have access to exact value of -Know only distribution of given - Not immediately useful since we want to compress P(,) =4 Encoding does not require f (y x): de-coupled from the encoded data ~ 13 Distributed Source oding Lossless compression of random variable - Intra coding: R H() (i) LP (e.g. DPM): Z=- R H( ) -Why DS? What are the potential applications? -How to encode the input exactly? (ii) DS: ~ -What is the coding performance? P(,) Encoding does not require de-coupled from the encoded data ~ 14 7

8 Why DS? When predictor is not available during encoding orrelated sources are not co-located Sensors network, wireless camera Low complexity video encoding Due to complexity constraint at the encoder time DS: ~ P(,) How to estimate this correlation? 15 Why DS? To decouple compressed data from predictor In LP, prediction residue is closely coupled with the predictor (i) LP (e.g. DPM): Z=- (ii) DS: ~ P(,) In DS, compressed data is computed from the correlation How to encode? More robust to uncertainty on or error in 16 8

9 DS Example Using oset takes value [0, 255] (uniformly) Intra coding requires 8 bits for lossless orrelation P(,): 4 > - -4 (i.e., - takes value in [-4,4)) an explore this correlation If is available at both the encoder and decoder LP: communicate the residue - P(,) Requires 3 bits to convey If is not available at the encoder, how can we communicate? 17 DS Example Using oset (ont d) How to communicate if is not available at the encoder? =2 P(,): 4 > A 1B 2 D E F G H A B D G 255 H Partition reconstruction levels into different cosets Each coset includes several reconstruction levels : transmit coset label (syndrome), need 3 bits : disambiguate label (closest to ) Requires 3 bits to convey 18 9

10 DS Example Using oset (ont d) an we use less than 3 bits? Try 2 bits =2 A B D A B D A B D. : Decoding error if we use less bits than required orrelation information determines the number of coset for error-free reconstruction If correlation were 2 > - -2 (i.e.,, are more correlated), 2 bits would work 19 DS - Main Ideas Encoding: Send ambiguous information to achieve compression E.g., one label represents group of reconstruction levels Decoding: information is disambiguated at the decoder using side information Pick the coset member closest to Level of ambiguity (hence the encoding rate) is determined by correlation between and More correlated, more ambiguous representation using less bits 20 10

11 DS - Main Steps Partition input into cosets Members in the coset are separated by minimum distance Send coset index At decoder, pick the member closest to side information in the coset Similar steps for advanced algorithm based on error correction code (E.g., LDP) How about the coding efficiency? 21 DS Theoretical Performance LP Lossless compression of random variable R H( ) DS [Slepian, Wolf; 1973] R H( ) P(,) Efficient encoding possible even when encoder does not have precise knowledge of 22 11

12 DS Properties - Inherent Error Resilience Even is corrupted by noise, it is still possible to reconstruct exactly Prevent error propagation : : +N ompressed data (i.e., coset label) is decoupled from 23 DS Properties Robust to Predictor Uncertainty Multiple predictor candidates at the decoder (e.g., multiview video) does not know exactly which predictor is available at decoder : : 24 12

13 Application Scenario 25 Application Scenario Video Transmission - Resilience to transmission error - In practical applications, we have to estimate the noise power at the encoder, P(N) + N corrupted by noise N [Sehgal, Jagmohan, Ahuja; IEEE Trans. Multimedia 04], [Majumdar, Wang, Ramchandran, Garudadri; PS 04], [Rane, Girod; VIP 06], 26 13

14 Application Scenario Flexible Decoding Flexible decoding, e.g., multiview video - DS is robust to predictor uncertainty - estimates correlation between and k - Recover exactly no matter which k is at decoder 0, 1, 2, 0 or 1 or 2 ( does not know which one) [heung, Ortega; MMSP 07] 27 Application Scenario Low omplexity Video Encoding Low complexity video encoding - Finding to predict is complex - P(,) can be estimated in low complexity, hopefully P(,) [Puri, Ramchandran, Allerton 02] [Aaron, Zhang, Girod, Asilomar 02], 28 14

15 Outline Introduction Motivation Source coding with decoder side-information only Simple example to illustrate DS idea Application scenarios Basic information-theoretic results Slepian-Wolf Wyner-Ziv Practical encoding/decoding algorithm Role LDP based Applications Low-complexity video encoding Scalable video coding Flexible decoding, e.g., multiview video 29 Slepian-Wolf Theorem Lossless compression of correlated sources: ={}, ={} Random variables, jointly distributed P(,) Possible to achieve total rate as low as H(,) - theoretical limit when the encoders can cooperate side-information is a special case Distributed source coding with decoder side-information: R H() R H( ) R H()+H( )=H(,) R R R=R +R H(,) 30 15

16 Slepian-Wolf Theorem (cont ) R R General case: R R H( ) R H( ) R H(,) H() H( ) Achievable rate region side-information H( ) R R=R +R =H(,) For video/image applications, usually operate at corner points (decoder side-information) 31 Wyner-Ziv Theorem Lossy compression of ={}, ={}: memoryless sources is available only at the decoder R R (D) ^ R R WZ (D) ^ P(,) In general, there is rate loss: R WZ (D)-R (D)

17 Wyner-Ziv Theorem Example Example: quadratic Gaussian case ={}, ={};, are jointly Gaussian Distortion metric is MSE No rate loss with Wyner-Ziv coding R WZ (D) = R (D) No rate loss in several other cases 33 Practical Wyner-Ziv oding Practical Wyner-Ziv coding: Quantizer Q Slepian-Wolf Slepian-Wolf Q Minimumdistortion Reconstruction ^ P(,) Lossy qunatization Lossless compression of quantization index Q using SW encoder At decoder, side information is used in: Slepian-Wolf decoding Reconstruct in the quantization bin specified by Q To achieve Wyner-Ziv limit, need Good quantization: E.g., TQ Good Slepian-Wolf code: E.g., based on LDP 34 17

18 Practical Slepian-Wolf ode onstruction Slepian-Wolf encoder plays a similar role as entropy coding in conventional source coding problem Transform T[] Quantizer Q Entropy coding Bit-stream Transform T[] Quantizer Q Slepian-Wolf Bit-stream P(,) Lossless compression Rely on transform, quantization to achieve good (lossy) coding performance 35 SW oding with Error orrecting ode Linear (n, k) binary error-correcting code k bits input to channel encoder k n bits output at channel encoder n-k bits redundancy hannel n ~ Defined by parity check matrix H Space of n-bits binary vector: For any n-bits binary vector A A H T B = S, S is syndrome (n-k bits vector) B A B If is one of the 2 k codeword, S=0 In general, 2 k different have the same S Partition space of into 2 n-k cosets Each coset has 2 k members H T = S 2 n members 2 k members In each coset, the minimum (Hamming) distance between the members is the same 38 18

19 LDP Based Slepian-Wolf ode ={}, ={}, to compress a n-bits binary vector Similar idea as the coset example we have mentioned : - Partition input space into cosets: - Send coset index: Space of n-bits vector: A B D E F G H A B D A B B A A B mod m = coset index H T = S 2 n-k cosets : - Use to disambiguate the information: - ompression performance: (n-k)/n - Number of cosets (min. distance between members of coset) depends on correlation 39 LDP Based Slepian-Wolf ode (cont ) : SW : H T = S SW : :??????? p( i i ) would use the SI to estimate the probability that each bit is being zero or one 40 19

20 Outline Introduction Motivation Source coding with decoder side-information only Simple example to illustrate DS idea Application scenarios Basic information-theoretic results Slepian-Wolf Wyner-Ziv Practical encoding/decoding algorithm Role LDP based Applications Low-complexity video encoding Scalable video coding Flexible decoding, e.g., multiview video 42 Low omplexity Video Encoding Shift motion estimation to decoder Low complexity video encoder High complexity decoder Application: uplink video transmission Video sensor network Mobile phone video Important work: [Puri, Ramchandran; 2002] [Aaron, Zhang, Girod; 2002] 43 20

21 Low omplexity Video Encoding - Example [Puri, Ramchandran; 2002], [Puri, Majumdar, Ramchandran; 2007] DS requires only P(,) at encoder : current macroblock to be encoded : best motion-compensated predictor from reference frame for Two problems: At encoder, how to estimate P(,) without searching? For low complexity encoding, avoid ME at encoder At decoder, how to obtain to decode? onventional ME requires to locate is not available time P(,) 44 Low omplexity Video Encoding Example (cont ) Block based : Transform, Quantization, Syndrome coding time P(,) Estimate P(,): Squared difference (SD) between co-located block in previous frame Use SD to infer P(,) thro classifier Require no motion search [Puri, Majumdar, Ramchandran; IEEE Trans. IP 2007] 45 21

22 Low omplexity Video Encoding Example (cont ) : x Require (best motioncompensated predictor) to perform Slepian-Wolf decoding onventional ME requires to locate But is not available sends also R of At decoder, try all possible candidate predictors For each candidate predictor, SW decoder outputs one decoded sequence closest to the candidate If the decoded sequence passes R test, declare current decoded sequence as Otherwise, try another candidate 46 Low omplexity Video Encoding oding Efficiency ompared to H.263+, FE, Intra-refresh 3dB loss when no packet drop 2dB gain at 10% packet drop rate [Puri, Majumdar, Ramchandran; IEEE Trans. IP 2007] 47 22

23 DS Application to Scalable Video oding 48 Some Work on Scalable oding Based on DS [u, iong; VIP 04, IEEE Trans. IP 06] Embedded enhancement layer similar to MPEG-4/H.26L FGS Robust to error in base layer Do not suffer performance loss due to layering [Tagliasacchi, Majumdar, Ramchandran, Tubaro; PS 04, Eurasip SP 06] Spatial/temporal/SNR scalability Based on PRISM [Puri, Ramchandran; Allerton 02] Robust to channel losses Flexible distribution of complexity [Sehgal, Jagmohan, Ahuja; PS 04] Multiple Wyner-Ziv encoded versions for different possible predictors streams an appropriate encoded version based on knowledge of predictors available at decoder [Wang, heung, Ortega; PS 04, Eurasip JASP 06] Improvement of MPEG4-FGS: Utilizing EL reconstruction for prediction Low complexity overhead: Avoid replicating EL reconstruction at encoder 49 23

24 Error Robust Scalable oding [u, iong; VIP 04, IEEE Trans. IP 06] Base layer (BL): Standard video codec Side Information (SI): BL reconstruction Enhancement layer (EL) based on bit-plane coding and DS Error corrupted base layer (SI) may still be used for DS decoding 50 Robust to Transmission Error DS-based system can achieve substantial improvement in situations with transmission error [u, iong; VIP 04, IEEE Trans. IP 06] Football 1% macroblock loss at BL BL: 1450 kbps EL: 200 kbps DS FGS In addition, the system does not suffer performance loss due to layering, i.e., same performance as the monolithic WZ coding 51 24

25 DS Application to Flexible Video Decoding 67 Flexible Video Decoding Apply DS to generate a single bitstream that can be decoded in several different ways View View Free viewpoint switching in multiview video compression Forward and backward video playback Robust video transmission Time 68 25

26 Flexible Decoding Example Free Viewpoint Switching in Multiview Video User selects a particular view: View [heung, Ortega; PS 07] User switches between different views during playback: Time video frame Decoding path In order to address viewpoint switching, compression schemes have to support multiple decoding paths 69 Free viewpoint switching poses challenges to multiview compression When users can choose among different decoding paths, it is not clear which previous reconstructed frame will be available to use in the decoding View Time Multiple decoding paths Either 0 or 1 or 2 will be available at decoder Uncertainty on predictor status at decoder!!!! 70 26

27 Problem Formulation To support low-delay free viewpoint switching (flexible decoding), encoder needs to operate under uncertainty on decoder predictor [heung, Ortega; MMSP 07] View does not know which i will be available at decoder Time ^ Assume feedback is not available Low-delay, interactive application Offline encoding of multiview data 71 Address Viewpoint Switching/Flexible Decoding Within losed-loop Predictive (LP) oding Framework has to send multiple prediction residues to the decoder View Prediction residue is tied to a specific predictor Time ^ ^ ^ Z Z 0, Z 1, Z 2 1 = Z 0 =- 0 Z 2 =- 2 ^ ^ =2 +Z2 ^ Z i : P-frame or SP-frame Overhead increases with the number of predictor candidates ^ may not be identical when different i are used as predictor - drifting 74 27

28 Solution Based on Distributed Source oding 78 DS - Virtual ommunication hannel Perspective In DS, encoder can communicate by sending parity information (E.g., [Girod, Aaron, Rane, Rebollo-Mondero; Proc. IEEE 04]) LP: DS: Enc Z=- Dec ^ Z + Dec ^ Parity information Parity information is independent of a specific predictor - What matters is the amount of parity information 79 28

29 Address Viewpoint Switching (Flexible Decoding) Using DS Under predictor uncertainty, encoder can communicate by sending an amount of parity corresponding to the worst case correlation View Time ^ N virtual channels: Z Dec ^ Z Dec ^ + Z 2 2 Dec ^ Parity corresponding to the worst case correlation noise 80 Viewpoint Switching (Flexible Decoding) LP vs. DS LP R LP = Σ H(Z i ) H(Z 0 ) H(Z 1 ) H(Z 2 ) View Time LP: ^ ^ ^ Z 0, Z 1, Z DS: Worst case parity ^ R DS = max ( H(Z i ) ) DS H(Z 0 ) H(Z 1 ) Bits required to communicate with i at decoder : H( i ) = H(Z i ) H(Z 2 ) 81 29

30 Parity information is independent of a specific predictor b(l) : SW : SW : b(l) :??????? p(b(l) i, b(l+1), b(l+2), ) 82 Experimental Results - Multiview Video oding Allow switching from adjacent views: three predictor candidates Ballroom (320x240, 25fps, GOP=25) View DS LP Time PSNR Intra Intra Inter Proposed Bit-rate (kbps) Our proposed algorithm out-performs LP and intra coding 88 30

31 Experimental Results Akko&Kayo (320x240, 30fps, GOP=30) 38 PSNR DS LP Intra Intra Inter Proposed Bit-rate (kbps) 89 Drifting Experiments Our proposed algorithm is almost drift-free, since quantized coefficients in DS coded MB are identically reconstructed LP DS Akko&Kayo (GOP=30) PSNR w ithout sw itching sw itching Drifting in LP PSNR w ithout sw itching sw itching Switching occurs Frame no Switching occurs Frame no. View Time View switching occurs at frame number

32 Scaling Experiments Number of coded bits vs. number of predictor candidates Bits per frame Intra v-1 v v DS LP Intra Inter Proposed Number of predictor candidates Bit-rate of DS-based approach increases at a slower rate compared with LP An additional candidate incurs more bits only if it has the worst correlation among all candidates 91 Summary: DS Application to Viewpoint Switching/Flexible Decoding DS-based coding algorithm to address viewpoint switching/flexible decoding Single bitstream to support multiple decoding paths Parity information independent of a specific predictor Overhead depends on the worst correlation rather than the number of decoding paths Outperform LP and intra coding in terms of coding performance Our proposed system is almost drift-free 93 32

33 References Introduction B. Girod, A. Aaron, S. Rane, and D. Rebollo-Monedero, Distributed video coding, Proceedings of the IEEE, Special Issue on Advances in Video oding and Delivery 93, pp , Jan Z. iong, A. Liveris, and S. heng, Distributed source coding for sensor networks, IEEE Signal Processing Magazine 21, pp , Sept Recent Advances Guillemot,., Pereira, F., Torres, L., Ebrahimi, T., Leonardi, R., Ostermann, J., Distributed Monoview and Multiview Video oding, Signal Processing Magazine, IEEE, Volume 24, Issue 5, Sept. 2007, Page(s):67-76 Workshop on Recent Advances in Distributed Video oding