ENSC Multimedia Communications Engineering Topic 4: Huffman Coding 2

Transcription

1 ENSC Multimedia Communications Engineering Topic 4: Huffman Coding 2 Jie Liang Engineering Science Simon Fraser University JieL@sfu.ca J. Liang: SFU ENSC 424 1

2 Outline Canonical Huffman code Huffman encoding Huffman decoding Limitations of Huffman code Context-adaptive Huffman coding J. Liang: SFU ENSC 424 2

3 Canonical Huffman Code Huffman codes for a given data set is not unique Huffman algorithm is needed only to compute the optimal codeword lengths Canonical Huffman code is well structured Given the codeword lengths, can find a canonical Huffman code J. Liang: SFU ENSC 424 3

4 Canonical Huffman Code Rules: Assign 0 to left branch and 1 to right branch Build the tree from left to right in increasing order of depth Each leaf is placed at the first available position Example: Codeword lengths: 2, 2, 3, 3, 3, 4, 4 Verify that it satisfies Kraft-McMillan inequality A non-canonical example N i= 1 2 The Canonical Tree l i J. Liang: SFU ENSC 424 4

5 Canonical Huffman Properties: The first code is a series of 0 Codes of same length are consecutive: 100, 101, If we pad zeros to the right side such that all codewords have the same length, shorter codes would have lower value than longer codes: 0000 < 0100 < 1000 < 1010 < 1100 < 1110 < Formula from level n to level n+1: C(n+1, 1) = 2 ( C(n, last) + 1): append a 0 to the next available level-n code First code of length n+1 Last code of length n If from length n to n + 2 directly: e.g., 1, 3, 3, 3, 4, C(n+2, 1) = 4( C(n, last) + 1) J. Liang: SFU ENSC 424 5

6 Advantages of Canonical Huffman 1. Reducing memory requirement Non-canonical tree needs: All codewords Lengths of all codewords Need a lot of space for large table Canonical tree only needs: Min: shortest codeword length Max: longest codeword length Distribution: Number of codewords in each level Min=2, Max=4, # in each level: 2, 3, 2 J. Liang: SFU ENSC 424 6

7 Advantages of Canonical Huffman 2. Simplifying the decoding See Hirschberg & Lelewer s paper for details (Not required in this course) Final note about Canonical Huffman: Different conventions of canonical form exist: Can be built from the right to the left (exchanging 0 and 1). This convention is used in, e.g., Practical Huffman coding at J. Liang: SFU ENSC 424 7

8 Outline Review of last lecture Huffman code Canonical Huffman code Huffman Encoding Huffman Decoding Limitations of Huffman code Context-adaptive Huffman coding J. Liang: SFU ENSC 424 8

9 Computer Implementation Code: a: 01, b: 1, c: 000, d: 0010, e: 0011 How to store the Huffman table? Need codewords: 8 bits, 16 bits, 32 bits unsigned char Codewords[5] = {1, 1, 0, 2, 3}; Also need to store length of each codes unsigned char Codelength[5] = {2, 1, 3, 4, 4}; Computer does not know the number of leading zeros. 1 and 01 are different codes, 0 and 000 are different codes Decoder also needs to store this table. J. Liang: SFU ENSC 424 9

10 Encoding For each input symbol End Find its codeword Output all bits of the codeword Needs bit-level operations: byte pointer, bit pointer Source alphabet A = {a, b, c, d, e} Probability distribution: {0.2, 0.4, 0.2, 0.1, 0.1} Code: a: 01, b: 1, c: 000, d: 0010, e: 0011 Sequence: abaddecade Compressed bitstream: bits: 3 bits / symbol ASCII: 10 bytes (80 bits) Compression ratio: 2.67 : 1 J. Liang: SFU ENSC

11 Outline Review of last lecture Huffman code Canonical Huffman code Huffman Encoding Huffman Decoding Direct Approach Table Look-up Method Multilevel Table Look-up Method Limitations of Huffman code Context-adaptive Huffman coding J. Liang: SFU ENSC

12 Huffman Decoding Direct Approach: Read one bit, compare with all codewords Slow Binary tree approach: Embed the Huffman table into a binary tree data structure Read one bit: if it s 0, go to left child. If it s 1, go to right child. Decode a symbol when a leaf is reached. Still a bit-by-bit approach J. Liang: SFU ENSC

13 Table Look-up 00 1 N: # of codewords L: max codeword length Expand to a full tree: Each Level-L node belongs to the subtree of a codeword. Equivalent to dividing the range [0, 2^L] into N intervals, each corresponding to one codeword. Read L bits, and find which internal it belongs to: Bar [4] = {010, 011, 100, 1000}; x = ReadBits (3); For (k = 0; k <= 3) { } if (x < Bar [k]) {Decode codeword k; break;} Still needs conditional operators: slow J. Liang: SFU ENSC

14 Table Look-up Method a: 00 b: 010 c: 011 d: 1 char HuffDec[8][2] = { }; { a, 2}, { a, 2}, { b, 3}, { c, 3}, { d, 1}, { d, 1}, { d, 1}, { d, 1} a b c J. Liang: SFU ENSC d x = ReadBits(3); k = 0; //# of symbols decoded While (not EOF) { } symbol[k++] = HuffDec[x][0]; length = HuffDec[x][1]; x = x << length; newbits = ReadBits(length); x = x newbits; x = x & 7;

15 Multi-level Table Look-Up Look-up table size: 2^L L=8: 256 entries L=16: entries! Most codes can be decoded by looking at a few bits Long codes rarely appear Multi-level Table Look-Up Divide-and-conquer One for frequently appeared codewords Others for longer codewords entries 110 L = 8 16 entries One-level: 256 entries Two-level: 32 entries! J. Liang: SFU ENSC

16 More Complicated Cases More than one hole in the first table: Need one 2 nd -layer table for each hole. Table Table 2 Table 3 J. Liang: SFU ENSC

17 Outline Huffman code Canonical Huffman code Huffman Decoding Limitations of Huffman Code Context-adaptive Huffman coding J. Liang: SFU ENSC

18 Limitations of Huffman Code Need a probability distribution Usually estimated from a training set But the practical data could be quite different Hard to adapt to changing statistics Must design new codes on the fly Context-adaptive method still need predefined tables Minimum codeword length is 1 bit Serious penalty for high-probability symbols Example: Binary source, P(0)=0.9 Entropy: -0.9*log2(0.9)-0.1*log2(0.1) = bit Huffman code: 0, 1 Avg. code length: 1 bit More than 100% redundancy!!! Joint coding is not practical for large alphabet. J. Liang: SFU ENSC

19 Outline Huffman code Canonical Huffman code Huffman Decoding Limitations of Huffman Code Context-adaptive Huffman coding J. Liang: SFU ENSC

20 Context-adaptive Huffman code Switch Huffman table based on previously encoded information (context) Decoder follows the same rules. Example: Binary source with P(X2i, X2i+1): P(0, 0) = 3/8, P(0, 1) = 1/8 P(1, 0) = 1/8, P(1, 1) = 3/8 Encode 2 symbols together Code from previous example: 1, 00, 010, and 1 tend to appear in cluster: x0 x1 x2 x3 x4 x5 x6 x7 Context (0,0) (1,1) (0,1) (1,0) (0,0) Use last two symbols (context) to predict the next two. Design one set of Huffman code for each possible context value. Used in H.264 (revisited later) J. Liang: SFU ENSC Codewords (0,1) 10 (1,0) (1,1)

21 Summary Huffman code generation: Sort, merge assign code Canonical Huffman code Left to right, short to long, take first valid leaf Huffman Decoding Direct, tree, table look-up, multi-level table look-up Limitations of Huffman Coding Adaptive is hard, not efficient Next: Golumb-Rice coding Arithmetic coding J. Liang: SFU ENSC