ENSC 424 - Multimedia Communications Engineering Huffman Coding () Jie Liang Engineering Science Simon Fraser University JieL@sfu.ca J. Liang: SFU ENSC 424
Outline Entropy Coding Prefix code Kraft-McMillan inequality Huffman Encoding Minimum Variance Huffman Coding Extended Huffman Coding J. Liang: SFU ENSC 424 2
Entropy Coding Design the mapping from source symbols to codewords Lossless mapping Goal: minimizing the average codeword length Approach the entropy of the source J. Liang: SFU ENSC 424 3
Example: Morse Code Represent English characters and numbers by different combinations of dot and dash (codewords) Examples: E I A T O S Z Problem: Letters have to be separated by space, Or paused when transmitting over radio. SOS: pause Not uniquely decodable! J. Liang: SFU ENSC 424 4
Entropy Coding: Prefix-free Code No codeword is a prefix of another one. Can be uniquely decoded. Also called prefix code Example:,,, Binary Code Tree Root node Internal node leaf node Prefix-free code contains leaves only. How to express the requirement mathematically? J. Liang: SFU ENSC 424 5
Kraft-McMillan Inequality Let C be a code with N codewords with length li, i=, N. If C is uniquely decodable, then N i= 2 If a set of li satisfies the inequality above, then there exists a prefix-free code with codeword lengths li, i=, N. l i J. Liang: SFU ENSC 424 6
Kraft-McMillan Inequality To see this, expand the binary code tree to depth L = max(li) N i= 2 l i J. Liang: SFU ENSC 424 7 N i= Number of nodes in the last level: Each code has a sub-tree: 2 L l The number of offsprings in the last level: K-M inequality: L = 3 i L 2 # of L-th level offsprings of all codes is less than 2^L. 2 L 2 L l i Leads to more than 2^L offspring
Outline Entropy Coding Prefix code Kraft-McMillan inequality Huffman Encoding Minimum Variance Huffman Coding Extended Huffman Coding J. Liang: SFU ENSC 424 8
Huffman Coding A procedure to construct optimal prefix-free code Result of David Huffman s term paper in 952 when he was a PhD student at MIT Shannon Fano Huffman (925-999) Observations: Assign short codes to frequent symbols. In an optimum prefix-free code, the two codewords that occur least frequently will have the same length. truncate a b a b J. Liang: SFU ENSC 424 9
Huffman Code Design Another property of Huffman coding: The codewords of the two lowest probability symbols differ only in the last bit. Requirement: The source probability distribution (Not available in most cases) Procedure:. Sort the probability of all source symbols in a descending order. 2. Merge the last two into a new symbol, add their probabilities. 3. Repeat Step, 2 until only one symbol (the root) is left. 4. Code assignment: Traverse the tree from the root to each leaf node, assign to the top branch and to the bottom branch. J. Liang: SFU ENSC 424
Example 3.2. Source alphabet A = {a, a2, a3, a4, a5} Probability distribution: {.2,,.2,.,.} Sort merge Sort merge Sort merge Sort merge a2 ().6 a(.2) a3(.2) a4(.).2.2.2.2.2.6 a5(.) Assign code J. Liang: SFU ENSC 424
Huffman code is prefix-free All codewords are leaf nodes No code is a prefix of any other code. (Prefix free) J. Liang: SFU ENSC 424 2
Average Codeword Length vs Entropy Source alphabet A = {a, b, c, d, e} Probability distribution: {.2,,.2,.,.} Code: {,,,, } Entropy: H(S) = - (.2*log2(.2)*2 + *log2()+.*log2(.)*2) = 2.22 bits / symbol Average Huffman codeword length: L =.2*2+*+.2*3+.*4+.*4 = 2.2 bits / symbol In general: H(S) L < H(S) + J. Liang: SFU ENSC 424 3
Huffman Code is not unique Two choices for each split:, or,.6.6.6.2.2.6 Multiple ordering choices for tied probabilities a.6 b.6 b.6 a.6 c.2 c.2 J. Liang: SFU ENSC 424 4
Minimum Variance Huffman Code Put the combined symbol as high as possible in the sorted list Prevent unbalanced tree: Reduce memory requirement for decoding (revisited later) Repeat previous example Compute average codeword length J. Liang: SFU ENSC 424 5
Extended Huffman Code Code multiple symbols jointly Composite symbol: (X, X2,, Xk) Alphabet increased exponentioally: N k Code symbols of different meanings jointly JPEG: Run-level coding H.264 CAVLC: context-adaptive variable length coding # of non-zero coefficients and # of trailing ones Revisited later J. Liang: SFU ENSC 424 6
Example Joint probability: P(X2i, X2i+) P(, ) = 3/8, P(, ) = /8 P(, ) = /8, P(, ) = 3/8 P(Xj = ) = P(Xj = ) = /2 Entropy H(Xj) = bit / symbol Joint Prob P(X2i, X2i+) X2i+ X2i 3/8 /8 /8 3/8 Second order entropy: H X 2 i, X 2i+ ) =.83 bits ( / 2 symbols, or.956 bits / symbol Huffman code for Xj:, Average code length Huffman code for (X2i, X2i+): bit / symbol :, :, :, : Average code length:.9375 bit /symbol J. Liang: SFU ENSC 424 7
Summary Goal of entropy coding: Reduce the average codeword length (the entropy is the lower bound) Prefix-free code: uniquely decodable code Kraft-McMillan Inequality: Characteristic of prefix-free code Huffman Code: Optimal prefix-free code Minimum variance code Next: Canonical Huffman Encoding and decoding J. Liang: SFU ENSC 424 8