Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Lecture 6 Review of Lossless Coding (II) May 28, 2009
Outline Review Manual exercises on arithmetic coding and LZW dictionary coding 1
Review
Lossy coding followed by lossless coding A/D Conversion Color Space Conversion Pre-Filtering Partitioning Predictive Coding Differential Coding Motion Estimation and Compensation Context-Based Coding Input Image/Video Pre- Processing Lossy Coding Lossless Coding Post- Processing (Post-filtering) Quantization Transform Coding Model-Based Coding Entropy Coding Dictionary-Based Coding Run-Length Coding Encoded Image/Video 3
Why do we need a new coding algorithm? Problems with Huffman coding Each symbol in X is represented by at least one bit. Coding for X n : A Huffman tree with X n nodes needs to be constructed. The value of n cannot be too large. Encoding with a Huffman tree is quite easy, but decoding can be difficult especially when X is large. Dynamic Huffman coding is slow due to the update of the Huffman tree from time to time. Solution: Arithmetic coding Encode a number of symbols (can be a message of a very large size n) incrementally (progressively) as a binary fraction in a real range [low, high) [0,1). 4
Arithmetic coding: An example X={A,B,C,D}, P=[0.6,0.2,0.1,0.1]. Input: x=acd Output: y [0.534, 0.54) 5
Arithmetic coding: Which fraction? Which fraction in [p c (x), p c (x)+p(x)) should be chosen? One proper choice: 1/2 l p(x)/2 l = dlog 2 (1/p(x))e +1 choose f(x) to be a binary integer i such that (i- 1)/2 l p c (x)<i/2 l <p c (x)+p(x) H b (P) L<H b (P)+2/n p c (x) i/2 l 0 1 Assuming the above choice is used to design a code f:x n Y*, it can be proved that f is a PF code. n can be as large as the size of the input message x. ) p(x) p c (x)+p(x) What is this? 6
Arithmetic coding: Termination problem Termination of the original message x Transmit the size of x to the decoder separately. Add an end of message symbol to X. Termination of the encoded bitstream f(x) Transmit the number of bits to the decoder separately. Use the choice of fraction introduced in last slide. Add an end of message symbol to X. 7
Arithmetic coding: Performance Arithmetic coding: H b (P) L<H b (P)+2/n Huffman coding: H b (P) L<H b (P)+1/n Arithmetic coding < Huffman coding? Theoretically yes if all the entropy coding schemes are working on the same source (X n,p n ). Practically no The complexity of almost all other entropy codes (including Huffman code and Shannon code) will soon reach its computational limit when n. But arithmetic coding works in an incremental way, so n can be as large as the size of the to-beencoded message itself. Arithmetic coding > Huffman coding! 8
Arithmetic coding: Noticeable features Non-block coding (Incremental coding) The output of an encoder is not the concatenation of codewords of consecutive symbols. Range coding The encoded message is actually a range in [0,1]. Separation between coding and the source s statistics The probability is used to divide a range with the coding process (instead of before it). Arithmetic coding is inherently dynamic/adaptive/contextbased coding. Patent-Cumbered A variant called Range Coding is patent-free. 9
Arithmetic coding: Practical issues Finite-precision problem Approximating probabilities and the arithmetic operations in finite-precision loss of coding efficiency, but negligible if the precision is high enough. Common bits of the two end points can be removed immediately and sent out. A renormalization process is needed to re-calculate the new values of the two end points. Carry-over/propagation problem A carry (caused by the probability addition) may propagate over q bits (such as 0.011111110 0.100000000 ), so an extra register has to be used to track this problem. 10
Binary arithmetic coding Solution to carry-over problem X=Y={0,1}, P=[p 0,1-p 0 ] n-bit finite precision is used: initially, [0,1) [0,2 n -1) Update the range as usual: low =low or low+(high-low)*p 0, high =low+(high-low)*p 0 or high. Do nothing if high-low 2 n-1 Do renormalization when high-low<2 n-1 When low, high<2 n-1 output a 0-bit, and [0,2 n-1 ) [0,2 n -1). When low, high 2 n-1 output a 1-bit, subtract 2 n-1 from low, high, and [2 n-1,2 n -1) [0,2 n -1). low<2 n-1 (low=01 ) but high 2 n-1 (high=10 ) store an unknown bit (which is the complement of the next known bit) in the buffer, subtract 2 n-2 from low, high, and [2 n-2,2 n-1 +2 n-2 ) [0,2 n -1) After all symbols are encoded, terminate the process by sending out a sufficient number of bits to represent the value of low. 11
Binary arithmetic coding 12
Multiplication-free implementation For binary arithmetic coding, we can further approximate high-low α=¾ ( high-low=½~1) Then, the update of low and high can be approximated without doing the multiplication. The optimal value of α is actually depends on the probability p 0, and in most cases smaller than ¾ (typical value is ⅔ or around 0.7). When p 0 >1/(2α)=⅔, it is possible αp 0 >½>high-low, so high <low, which is a big mistake. Solution: dynamically exchange p 0 and p 1 such that p 0 <½ always holds. 0 = LPS (less/least probable symbol) and 1 = MPS (more/most probable symbol). 13
Dictionary coding: The basic idea Use a dictionary and encode a symbol or a sequence of symbols as the index of the dictionary. The dictionary can be static (such as in Digram coding) or dynamic (such as in LZ coding family). The dictionary can be implicit (such as in LZ77) or explicit (such as in LZ78). The dictionary can be considered as an (maybe very rough) approximation of the probability distribution of the source. 14
LZ77 LZ78 LZW LZ77 (Sliding window compression) A search buffer which includes a finite number of encoded is used as an implicit dictionary. Output: (o, l, S). LZ78 An explicit dictionary is constructed from empty by adding new symbols or sequences of symbols into it. Output: (i, S) LZW (Patent-Cumbered) An explicit dictionary is initialized to include all single symbols and then expanded to include sequences of symbols. Output: i. 15
1-D run-length coding Statistical model: Discrete Markov Source S={S 1,S 2 } with 4 transition probabilities: P(S i /S j ), i, j {1,2}. S 1 S 2 P(S 1 /S 1 ) P(S 2 /S 2 ) P(S 2 /S 1 ) P(S 1 /S 2 ) Runs: Repetition of symbols S 1 or S 2 How to code: (S i, Run1,Run2, ) All the run-lengths form a new random source, which can be further coded with a (modified) Huffman coder or a universal code. 16
How entropy coding is used Huffman/Arithmetic coding + Dictionary coding + Universal codes + Predictive coding Spatial difference tends to have a monotonic probability distribution Universal codes + Run-length coding The run-length pairs tends to have a monotonic probability distribution 17
Lossless image coding standards JBIG (Joint Bi-level Image Experts Group): QM-coder Lossless JPEG family: Huffman coding/arithmetic coding JBIG 2: MQ-coder GIF (Graphics Interchange Format): LZW PNG (Portable Network Graphics): DELATE = LZ77 + Huffman coding TIFF (Tagged Image File Format): RLE, LZW Compressed BMP and PCX: RLE 18
Manual Exercises
Test arithmetic coding X={A,B,C}, P={0.5,0.3,0.2} x= bcac Please try to find a fraction to as the encoded result of x. Please try to find the proper binary fraction according to the rule l = dlog 2 (1/p(x))e +1 20
Testing LZW X={a,b,c,d,e} Try to encode the following two messages and see how the dynamic dictionaries will be constructed. x= aaaaa x= abcdabc Reference: Khalid Sayood, Introduction to Data Compression, Section 5.4.2 The LZ78 Approach, pp. 130-133, 3rd Edition, Morgan Kaufmann, 2005 21